Adaptive On - Line Learning Algorithms for Blind Separation | Maximum Entropy and Minimum Mutual Information

There are two major approaches for blind separation: Maximum Entropy (ME) and Minimum Mutual Information (MMI). Both can be implemented by the stochastic gradient descent method for obtaining the de-mixing matrix. The MI is the contrast function for blind separation while the entropy is not. To justify the ME, the relation between ME and MMI is rstly elucidated by calculating the rst derivative of the entropy and proving that 1) the the mean-subtraction is necessary in applying the ME and 2) at the solution points determined by the MI the ME will not update the de-mixing matrix in the directions of increasing the cross-talking. Secondly, the natural gradient instead of the ordinary gradient is introduced to obtain e cient algorithms, because the parameter space is a Riemannian space consisting of matrices. The mutual information is calculated by applying the Gram-Charlier expansion to approximate probability density functions of the outputs. Finally, we propose an e cient learning algorithm which incorporates with an adaptive method of estimating the unknown cumulants. It is shown by computer simulation that the convergence of the stochastic descent algorithms is improved by using the natural gradient and the adaptively estimated cumulants.